Skip to content

[NVIDIA][AgentX] GB300 Muse Spark Open Dynamo+SGLang: retuned aggregated points and new TP8 disaggregated points / [NVIDIA][AgentX] GB300 Muse Spark Open Dynamo+SGLang:重新调优聚合配置点并新增 TP8 分离式配置点 - #4757

Merged
nvpohanh merged 5 commits into
mainfrom
musespark-fp4-gb300-dynamo-sglang-agentic-agg-disagg-1103
Nov 10, 2026

Conversation

@shyeh25

@shyeh25 shyeh25 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Moves the GB300 Muse Spark Open Dynamo + SGLang AgentX lane to lmsysorg/sglang:nightly-dev-cu13-20261103-797af717 (multiarch digest pinned) and runs the ai-dynamo 1.7.0.dev20261102 that the image ships (dynamo.install: false). It replaces the aggregated points under a new -agg key, adds TP8 disaggregated points under a new -disagg key, and removes musespark-fp4-gb300-dynamo-sglang-agentic-dflash (#4344).

Config key Topology Conc Nodes Key settings
musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg (new) TP4/EP4 46, 50, 56 1 HiCache 140 GB per rank (write_back, 0.80 DRAM utilization), fcfs, 32 running requests, prefill-decode interval 24 / 20 / 8, chunked prefill 16384, breakable prefill CUDA graph, draft attention trtllm_mha
TP8/EP8 36 2 GPU-resident KV, 36 running requests, shortest-prefill-first, prefill-decode interval 128, draft attention fa4
TP8/EP8 1 2 GPU-resident KV, 1 running request, prefill-decode interval 64, draft attention trtllm_mha
musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg (new) 3x TP8 prefill + 4x TP8 decode 672 14 56 requests per decode worker
2x TP8 prefill + 3x TP8 decode 352 10 40 requests per decode worker
1x TP8 prefill + 2x TP8 decode 160, 224 6 24 / 36 requests per decode worker
1x TP8 prefill + 3x TP8 decode 224 8 20 requests per decode worker
1x TP8 prefill + 4x TP8 decode 32, 56, 112 10 3 / 3 / 6 requests per decode worker
1x TP8 prefill + 5x TP8 decode 120 12 5 requests per decode worker
  • Disaggregated points share one setup: Mooncake KV transfer; prefill HiCache 150 GB per rank (write_back, 0.80 DRAM utilization), chunked prefill 32768, shortest-prefill-first; decode without layernorm SP. The frontend runs on the head node without KV events; the 3P4D and 2P3D points use soft session affinity across prefill workers.
  • Full-attention decode in trtllm_mha splits the KV into 16 chunks (SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS=16).
  • SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0 on every role.
  • Removes the nine config(musespark): add GB300 Dynamo SGLang AgentX aggregated points / config(musespark): 新增 GB300 Dynamo SGLang AgentX 聚合配置点 #4344 points (TP4 c8, c16, c20, c32, c40; TP8 c3, c4, c10, c14). Each removed point is exceeded on both throughput per GPU and mean interactivity by a new point; for example, 1P4D c56 exceeds TP8 c10 and c14.
AI model disclosure
  • Model/version: Claude Opus 5.5
  • Role: tuning, config changes
PR checklist and change type

Type of Change

  • Bug fix
  • New feature
  • Configuration change
  • Documentation update
  • Other (please describe)

Checklist

  • I have completed the AI model disclosure and kept it current
  • I have tested my changes locally
  • I have updated documentation if necessary
  • If python -m infx.matrix.changed --base origin/main lists affected config keys, I have appended one new entry covering them to the physical end of inferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entries
  • If this PR appends an inferencex-e2e/perf-changelog.yaml entry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs): full-sweep-fail-fast (recommended), full-sweep-enabled, or non-canary-full-sweep-enabled. Optional modifiers all-evals, evals-only, and agentx-fast require a primary label; the last two block reuse while applied.
  • Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.
中文

将 GB300 Muse Spark Open Dynamo + SGLang AgentX 配置迁移到 lmsysorg/sglang:nightly-dev-cu13-20261103-797af717(固定 multiarch digest),并直接使用镜像自带的 ai-dynamo 1.7.0.dev20261102(dynamo.install: false)。本 PR 在新的 -agg config key 下替换聚合配置点,在新的 -disagg config key 下新增 TP8 分离式配置点,并移除 musespark-fp4-gb300-dynamo-sglang-agentic-dflash(#4344)。

  • musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg(新增):
    • TP4/EP4 c46、c50、c56(1 个节点):每 rank HiCache 140 GB(write_back,DRAM 利用率 0.80),fcfs 调度,32 个运行请求,prefill-decode interval 分别为 24 / 20 / 8,chunked prefill 16384,prefill CUDA graph 使用 breakable,draft attention 使用 trtllm_mha。
    • TP8/EP8 c36(2 个节点):KV 留在 GPU,36 个运行请求,shortest-prefill-first 调度,prefill-decode interval 128,draft attention 使用 fa4。
    • TP8/EP8 c1(2 个节点):KV 留在 GPU,1 个运行请求,prefill-decode interval 64,draft attention 使用 trtllm_mha。
  • musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg(新增),prefill 与 decode worker 均为 TP8:
    • 3P4D c672(14 个节点),每个 decode worker 56 个请求。
    • 2P3D c352(10 个节点),每个 decode worker 40 个请求。
    • 1P2D c160、c224(6 个节点),每个 decode worker 24 / 36 个请求。
    • 1P3D c224(8 个节点),每个 decode worker 20 个请求。
    • 1P4D c32、c56、c112(10 个节点),每个 decode worker 3 / 3 / 6 个请求。
    • 1P5D c120(12 个节点),每个 decode worker 5 个请求。
  • 分离式配置点共用设置:Mooncake KV 传输;prefill 每 rank HiCache 150 GB(write_back,DRAM 利用率 0.80),chunked prefill 32768,shortest-prefill-first 调度;decode 关闭 layernorm SP。前端运行在 head 节点,不使用 KV event;3P4D 与 2P3D 在多个 prefill worker 之间使用 soft session affinity。
  • 全注意力层在 trtllm_mha decode 时把 KV 切成 16 段(SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS=16)。
  • 所有角色均设置 SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0。
  • 移除 config(musespark): add GB300 Dynamo SGLang AgentX aggregated points / config(musespark): 新增 GB300 Dynamo SGLang AgentX 聚合配置点 #4344 的九个配置点(TP4 c8、c16、c20、c32、c40;TP8 c3、c4、c10、c14)。每个移除的点在每 GPU 吞吐与平均交互性两个轴上都被某个新点超过;例如 1P4D c56 同时超过 TP8 c10 与 c14。

AI 模型使用说明:Claude Opus 5.5;负责调优与配置修改。

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

Comment on lines +57 to +63
PYTHONUNBUFFERED: "1"
SGLANG_TIMEOUT_KEEP_ALIVE: "900"
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: "1"
SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS: "1"
SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION: "0"
SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS: "16"
SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS: "4"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Muse Spark agentic runs could get turns rejected with HTTP 529 once a worker fills up. The base branch's Dynamo 1.6 did not have this backend admission check. The image's Dynamo 1.7 admits only a limited number of requests per engine, and almost every new point runs above max-running-requests. Fix: set DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT and DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT to "10000" on every engine role, which covers the 3 sites listed. Same pattern at 3 sites (agg-variants.yaml:54, disagg-variants.yaml:55, disagg-variants.yaml:113).

Why this was flagged

The trigger is load above what one engine has room for, under the Dynamo 1.7.0.dev20261102 that the image ships. The base branch installed the 1.6.0.dev20261016 wheel, but dynamo.install: false at agg-variants.yaml:9 and disagg-variants.yaml:9 now uses the image's 1.7 build. Every other dynamo-sglang 1.7 key in the repo raises these three limits to "10000", in dsv41flash agg-variants.yaml:69-73 and minimaxm31flash sglang agg-variants.yaml:63-65. Their comment says "Dynamo 1.7 backend admission rejects turns with 529 once the engine is full". The new points routinely run beyond that capacity. TP4 c46/c50/c56 cap max-running-requests at 32. Disagg decode capacity is 4x56=224 at c672. Rejected turns would fail the AgentX run or skew its throughput and interactivity results compared with the 1.6 runs on the base branch. Neither recipe sets any DYN_BACKEND_ADMISSION_* variable.

Verification: The diff changes the base's wheel: "1.6.0.dev20261016" to dynamo.install: false (agg-variants.yaml:9; also disagg-variants.yaml:9). Neither musespark recipe sets DYN_BACKEND_ADMISSION_*. Agg c46/c50/c56 run against max-running-requests: 32 on a single TP4 engine.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not applicable to this image. Dynamo 1.7.0.dev20261102 includes ai-dynamo/dynamo#17481 (merged 2026-10-31), which made backend admission opt-in: with no engine limit set, the gate is bypassed (no queueing, no 529). Every run of these recipes logs Backend admission configuration engine_request_limit=unlimited engine_request_waiting_limit=unlimited request_queue_limit=0, and no 529s occurred. The dsv41flash / minimaxm31flash recipes set 10000 because they targeted earlier 1.7 builds, where admission was on by default. Keeping the recipes as measured.

@shyeh25 shyeh25 added the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

@nvpohanh

nvpohanh commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs (musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg, musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg) will happen in sgl-project/sglang#48392, in the Agentic Long-Context Deployment section of the Muse Spark Open cookbook page.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (b37254d0c) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

shyeh25 and others added 2 commits November 4, 2026 04:44
…ted points and new TP8 disaggregated points

Move the GB300 Muse Spark Open Dynamo + SGLang AgentX lane to
lmsysorg/sglang:nightly-dev-cu13-20261103-797af717 (digest pinned) with
its bundled ai-dynamo 1.7.0.dev20261102. Rename the aggregated key to
musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg with TP4 c46/c50/c56
(HiCache 140 GB per rank) and TP8 c1/c36, add
musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg with nine TP8
prefill / TP8 decode points (c32 to c672), and remove
musespark-fp4-gb300-dynamo-sglang-agentic-dflash.

将 GB300 Muse Spark Open Dynamo + SGLang AgentX 迁移到
lmsysorg/sglang:nightly-dev-cu13-20261103-797af717(固定 digest),使用镜像自带的
ai-dynamo 1.7.0.dev20261102。聚合 key 改名为 -agg(TP4 c46/c50/c56,每 rank
HiCache 140 GB;TP8 c1/c36),新增 -disagg(九个 TP8 prefill / TP8 decode 点,
c32 至 c672),并移除原 key。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…G_TRTLLM_MHA_FULL_ATTN_KV_SPLITS

Drop SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS and
SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS from the aggregated and
disaggregated recipes; both fall back to the SGLang defaults (1 and 12).

聚合与分离式配方移除 SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS 与
SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS,改用 SGLang 默认值(1 与 12)。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shyeh25
shyeh25 force-pushed the musespark-fp4-gb300-dynamo-sglang-agentic-agg-disagg-1103 branch from b37254d to 704ebb2 Compare November 4, 2026 12:45
@nvpohanh

nvpohanh commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] sgl-project/sglang#48392 has been merged. This PR's AgentX configs (musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg, musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg) are now in the Agentic Long-Context Deployment section of the SGLang Muse Spark Open cookbook: https://docs.sglang.io/cookbook/autoregressive/Meta/MuseSparkOpen#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (b37254d0c) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

…X client to load its trace dataset

The recipe-level HF_HUB_OFFLINE=1 is propagated to the benchmark step, so
AIPerf cannot load semianalysisai/cc-traces-weka-062126 when its dataset
cache misses (OfflineModeIsEnabled). Set HF_HUB_OFFLINE=0 in benchmark.env
for the aggregated and disaggregated recipes; workers stay offline.

基准测试步骤会继承配方顶层的 HF_HUB_OFFLINE=1,数据集缓存未命中时 AIPerf
无法加载 semianalysisai/cc-traces-weka-062126(OfflineModeIsEnabled)。
聚合与分离式配方在 benchmark.env 设置 HF_HUB_OFFLINE=0,推理 worker 仍保持离线。

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

nvpohanh commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] The SGLang cookbook update of this PR's AgentX configs (musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg, musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg) will happen in sgl-project/sglang#48634, in the Agentic Long-Context Deployment section of the Muse Spark Open cookbook page. The configs have been live in that section since sgl-project/sglang#48392; sgl-project/sglang#48634 refreshes them with the pin below.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (7caf5718e) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

nvpohanh commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] sgl-project/sglang#48634 has been merged. This PR's AgentX configs (musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg, musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg) are now in the Agentic Long-Context Deployment section of the SGLang Muse Spark Open cookbook: https://docs.sglang.io/cookbook/autoregressive/Meta/MuseSparkOpen#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (7caf5718e) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

nvpohanh commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] The SGLang cookbook update of this PR's AgentX configs (musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg, musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg) will happen in sgl-project/sglang#48857, in the Agentic Long-Context Deployment section of the Muse Spark Open cookbook page. The configs have been live in that section since sgl-project/sglang#48392; sgl-project/sglang#48857 refreshes them with the pin below.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (7caf5718e) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

nvpohanh commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] sgl-project/sglang#48857 has been merged. This PR's AgentX configs (musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg, musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg) are now in the Agentic Long-Context Deployment section of the SGLang Muse Spark Open cookbook: https://docs.sglang.io/cookbook/autoregressive/Meta/MuseSparkOpen#agentic-long-context-deployment

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (7caf5718e) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

nvpohanh commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the pipeline of this PR. I'll keep monitoring it until it passes. If a job fails, I may rerun it when the cause is infrastructure, or push a fix or a merge of the latest main when the cause is in this PR.

@nvpohanh

nvpohanh commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] Hi @cquil11, could you help with reusing this PR's sweep?

Run Sweep 41854172505 on the current head 7caf5718 has every benchmark and eval job green: all 14 multi-node agentic / points, all 14 multi-node agentic eval / jobs, collect-results, collect-evals and compare-results. Results are on the dashboard.

The run is still marked failure because of one job, calc-success-rate. It lists the run's jobs with GET /actions/runs/41854172505/jobs?filter=all, and GitHub returns HTTP 502 for page 2 and later of that list. I reran the failed job six times (attempts 9–14), and it failed the same way each time. Each rerun also adds about 50 job records to the list (now 700+), so further reruns only make it worse. I then checked those pages every 10 minutes for two hours (15:00–17:00 UTC), and they never loaded.

Since /use requires a run whose overall result is success, I haven't posted /use here. Could you let us know how you'd like to proceed, for example by accepting this run for reuse, or by re-triggering the sweep if you'd prefer?

To keep this from happening on future runs, #4868 makes calc-success-rate retry each page and fall back to the latest attempt when the full job history can't be listed.

Thank you!

@nvpohanh

nvpohanh commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] cc @functionstackx as well: could you take a look at the reuse question in my comment above, in case Cam is not available? Thank you!

…ynamo-sglang-agentic-agg-disagg-1103

Conflict in inferencex-e2e/perf-changelog.yaml resolved by keeping main's
entries and appending this PR's entry at the end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

nvpohanh commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] We're starting a fresh full sweep for this PR instead of reusing run 41854172505.

Why:

  • In that run, every benchmark and eval job passed (14 multi-node agentic / points and 14 multi-node agentic eval / jobs). It is still marked failure only because of calc-success-rate.
  • calc-success-rate lists the run's jobs from every attempt. After 14 attempts the list has 700+ entries, and GitHub returns HTTP 502 for page 2 and later of it. That was still the case on every check from 15:00 to 17:00 UTC yesterday, and again today.
  • Rerunning the failed job can't fix this, because each rerun only makes the job list longer. /use needs a run whose overall result is success, so the old run can't be pinned.
  • The calc-success-rate fix in [fix] Retry calc-success-rate job listing and fall back to the latest attempt / [fix] calc-success-rate 列出 job 时重试并回退到最新 attempt #4868 only applies to runs that start after it merges. A new run also has a much shorter job list.

What changed on the branch: I merged the latest main with a merge commit, not a rebase. The only conflict was the append-only tail of inferencex-e2e/perf-changelog.yaml, resolved by keeping main's entries and appending this PR's entry at the end. The PR's recipes and config entries are unchanged. This push starts the new sweep, using the existing full-sweep-fail-fast label.

@cquil11 @functionstackx: no action needed on my earlier reuse question. Sorry for the noise, and thank you!

@nvpohanh

nvpohanh commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] The SGLang cookbook update of this PR's AgentX configs (musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg, musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg) will happen in sgl-project/sglang#49260, in the Agentic Long-Context Deployment section of the Muse Spark Open cookbook page. The configs have been live in that section since sgl-project/sglang#48392; sgl-project/sglang#49260 refreshes them with the pin below.

This PR has no authorized /use <run_id> comment yet, so the cookbook shows the recipe at this PR's head (e465e9a21) as "in progress". A later cookbook PR switches it to the measured run once a run is pinned or this PR merges.

@nvpohanh

Copy link
Copy Markdown
Collaborator

/use 42374203350

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

@nvpohanh staged run 42374203350.

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

@nvpohanh

Copy link
Copy Markdown
Collaborator

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items
  • Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
  • Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
  • Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
  • Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
  • Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
  • Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
  • For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
  • Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
  • Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
  • Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
  • For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
  • If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
  • For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
  • If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
  • Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
    • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
  • Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
  • If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
  • If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
  • Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

  • Assessed head: e465e9a2199e4e6a397cc7ba59db9a24761c6194, the commit the pinned sweep ran on (no later commits). Checklist fetched from main at a72dd0136.
  • PR validation (pinned run): Run Sweep 42374203350 on e465e9a2, pinned with /use 42374203350. All 14 multi-node agentic / points (13 + the c1 canary) and all 13 multi-node agentic eval / jobs passed, as did collect-results, collect-evals and compare-results. The run's overall conclusion is failure only because the calc-success-rate bookkeeping job gets HTTP 502 from GitHub when listing the run's 649 job records across 13 attempts (jobs?filter=all, pages 4-6 of 100 time out); it runs and measures nothing. The 1P2D c224 point has a superseded attempt-1 copy (job cancelled after upload); the attempt-5 result is the one counted below.
  • Evals: 13 gsm8k evals on the same image digest, em_strict 0.9757-0.9788. Every row is ok against both its best sibling topology and the exact ingested Muse Spark GB300 dynamo-sglang cohort (run 39427144732, 0.9788), within the 2-sigma threshold.
  • Chat template: the AgentX replay client uses the chat endpoint: aiperf input_config.endpoint.type = chat in the c1 raw artifact agentic_musespark_p1x8ep8_d0x8ep8_conc1_* (aiperf_artifacts/profile_export_aiperf.json), served through the checkpoint's chat template with reasoning-parser: muse.
  • Draft precision (shipped: BF16, effective: BF16):
    • Draft: RadixArk/Muse-Spark-Open-DFlash2 at 46c90f412549c472941a68382d91a99e5e6372a0 (DFlash2DraftModel), the same on the agg, prefill and decode roles, with 8 draft tokens.
    • Stored precision: config.json has dtype: bfloat16 and no quantization_config; the repo holds only config.json and model.safetensors (no hf_quant_config.json); Hugging Face safetensors metadata reports 4,887,479,552 parameters, all BF16.
    • Pinned image default handling (SGLang 797af717): with no --speculative-draft-model-quantization, the draft inherits --quantization modelopt_fp4 (serving_hook.py#L677-L690). Because the draft checkpoint has no quantization metadata, this resolves to ModelOptFp4Config.for_online_weight_quantization (weight_utils.py#L382-L388). Its get_quant_method returns UnquantizedLinearMethod for every linear layer and quantizes only FusedMoE (nvfp4_online.py#L163-L188). The DFlash draft model has no FusedMoE layers and sets supports_quantization = False (dflash.py#L833-L847), so its weights stay BF16, as shipped.
    • No draft dtype or quantization override, checkpoint substitution, or patch. The fp8_e4m3 KV cache is the upstream-supported dtype, applied to target and draft alike.
    • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set in either recipe and does not appear in any of the 14 benchmark job logs; its default is False (environ.py#L1257).
  • Golden AL: injected by the harness (infx/srt_slurm/synthetic_acceptance.py) from muse-spark-open-816b-a42b-nvfp4__dflash__muse-spark-open-dflash2__default-default.yaml, thinking_on, 7 speculative tokens (DFlash verify block of 8 minus the current token): golden AL 4.05. signoff_facts golden-al returns matches: true. All 14 benchmark job logs show SGLANG_SIMULATE_ACC_LEN=4.05, SGLANG_SIMULATE_ACC_METHOD=match-expected and SGLANG_SIMULATE_ACC_TOKEN_MODE=real-draft-token on every serving role.
  • MODELS.md: musespark (added 2026-10-15) is active for agentic coding; no deprecated model, scenario, or combination is submitted.
  • Model architecture: no --hf-overrides, --json-model-override-args, or model-file edits. The target is the NVFP4 checkpoint served with modelopt_fp4.
  • Integer all-reduce: GB300 runs the CUDA path. The only integer all-reduce controls in the pinned SGLang are ROCm quick-reduce (ROCM_QUICK_REDUCE_QUANTIZATION, SGLANG_ROCM_QUICK_REDUCE_INT8_BELOW_MB); they cannot be reached on CUDA and are not set. No other all-reduce quantization setting appears in the recipes.
  • Dynamo TRT-LLM metrics: N/A. This PR affects only dynamo-sglang configs and has no TRT-LLM worker.
  • Images and ordering: lmsysorg/sglang:nightly-dev-cu13-20261103-797af717@sha256:b4323a896f349d43a267c419c0267538b06c64a73110a0d0fd87496234df5dcd (upstream lmsysorg; the digest resolves now). It is built from SGLang commit 797af7170f34, the main squash-merge of [Refactor] scheduler: Check request ownership once at step launch sgl-project/sglang#43904; compare 797af717...main shows main 557 commits ahead and 0 behind. Dynamo 1.7.0.dev20261102 is the version shipped in the image (dynamo.install: false). Only dynamo-sglang entries are added, so engine-first ordering holds.
  • Upstream recipe: the published SGLang cookbook Muse Spark Open page (Agentic Long-Context Deployment section), sourced from docs/src/snippets/agentx/meta-models/muse-spark-open.jsx. [Doc] cookbook: Refresh SemiAnalysis AgentX launches to InferenceX main 0d7f94e7 sgl-project/sglang#49260 (MERGED 2026-11-09) refreshed it to this PR's head e465e9a2. It covers both musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg and ...-dflash-disagg:
    • etcd and NATS services and the Dynamo frontend command (KV router, soft session affinity for 3P4D and 2P3D);
    • the agg, prefill and decode dynamo.sglang commands with the flags and environment variables listed in this PR.
    • Excluded as InferenceX-only: Slurm/sbatch/srun options, health checks, observability, Hugging Face cache path and offline mode, the aiperf client environment (AIPERF_*, TOKENIZER, AGENTIC_WARMUP_GRACE_PERIOD), and the harness-injected SGLANG_SIMULATE_ACC_*, which the cookbook discloses as synthetic acceptance.
  • No engine patches: the recipes have no setup script, patch, or pip install. The runner applies only the existing srt-slurm orchestrator patches (runners/srt-slurm/patches/573-participating-gpus.patch, 900-pool-expected-devices.patch) to the srt-slurm checkout and installs harness and client tooling. No waiver is needed.
  • append-only: N/A. The changelog entry does not use append-only: true.
  • AgentX stream interval: stream-interval: 20 on the agg role and on the prefill and decode roles; no override changes it.
  • Removed points: each of the 9 removed musespark-fp4-gb300-dynamo-sglang-agentic-dflash points (2026-10-22, old dev-muse-spark-open image) is dominated in both tput_per_gpu and mean_intvty by a measured head point (signoff_facts dominance: undominated_removed: []). For example, 1P4D c56 dominates TP8 c10 and c14.
  • Pareto coverage: one affected curve (Muse Spark Open / AgentX / GB300 dynamo-sglang / FP4 / run 42374203350 / P90 E2EL), with agg and disagg combined: 14 measured points from the dashboard's unofficial-run API for run 42374203350, after dropping the superseded 1P2D c224 copy. pareto_coverage reports 12 frontier points (13 at P75).
  • Code quality: a config and recipe change only: two schema-2 srt-slurm variant files, nvidia-master.yaml, and one changelog entry appended at the end. The master config and recipes agree on workers, TP/EP, and node math for all 14 points, and check-changelog passed.

Signed: nvpohanh

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @nvpohanh is listed on the specific owner lines for inferencex-e2e/configs/nvidia-master.yaml and /inferencex-e2e/benchmarks/*_node/srt-slurm-recipes/**/gb300-*/; perf-changelog.yaml falls only under the * catch-all.

✅ Check 1 (In-PR sweep + evals): PASS — Head e465e9a2 (the pinned SHA) has all 14 multi-node agentic / and canary jobs plus all 13 multi-node agentic eval / jobs green in Run Sweep 42374203350 (headSha e465e9a2). The run is marked failure only because calc-success-rate hit HTTP 502 listing jobs; it measures nothing. There are no commits after the pinned SHA.

✅ Check 2 (Eval accuracy): PASS — 13 gsm8k rows from run 42374203350 on lmsysorg/sglang:nightly-dev-cu13-20261103-797af717@sha256:b4323a89..., em_strict 0.9757-0.9788 (n=1319). Every row is ok against its best sibling and against the exact ingested cohort (run 39427144732, 0.9788); the largest gap is 0.0030 against a threshold of 0.0116. Each eval runs on its own point's server config, so every changed path was exercised.

✅ Check 3 (Upstream recipe): PASS — sgl-project/sglang#49260 is MERGED (2026-11-09). The published Muse Spark Open cookbook AgentX data (muse-spark-open.jsx) is generated from PR 4757 at e465e9a2. It covers both -agg (TP4 c46/50/56, TP8 c1/c36) and -disagg (1P2D/1P3D/1P4D/1P5D/2P3D/3P4D), with etcd/NATS, the Dynamo frontend (soft session affinity on 2P3D/3P4D), and prefill/decode dynamo.sglang commands whose flags and env match the recipes. The harness/observability exclusions are justified in the sign-off.

✅ Check 4 (Reuse command): PASS — nvpohanh (COLLABORATOR) posted /use 42374203350 on its own line (comment), naming the Check 1 run.

✅ Check 5 (Checklist template): PASS — All 20 items in the current PR_REVIEW_CHECKLIST.md template are present and checked.

✅ Check 6 (Upstream image / provenance / ordering): PASS — Both keys use upstream lmsysorg/sglang (digest-pinned). Its build commit 797af717 (sgl-project/sglang#43904) is on SGLang main (558 commits ahead, 0 behind). Only dynamo-sglang entries are added, so the engine-first ordering rule does not apply.

✅ Check 7 (Deprecated models): PASS — musespark (added 2026-10-15) is active for agentic coding in MODELS.md, and nothing deprecated is submitted.

✅ Check 8 (No architecture hacks): PASS — There are no --hf-overrides, --json-model-override-args, or model-file edits, and no accuracy-trading levers. On GB300 (CUDA) no quick-reduce or integer all-reduce control is set or reachable.

✅ Check 9 (Chat template): PASS — The c1 raw artifact's aiperf_artifacts/profile_export_aiperf.json records endpoint.type: chat.

✅ Check 10 (No engine patches): PASS — The recipes have no setup script, patch, pip install, or source edits. The only patches applied are the existing srt-slurm orchestrator patches; the engine image is untouched.

✅ Check 11 (Golden AL): PASS — DFlash with 8 draft tokens gives 7 speculative tokens, and signoff_facts golden-al returns 4.05 (matches: true). The 3P4D and TP4 c46 job logs show SGLANG_SIMULATE_ACC_LEN=4.05, match-expected, and real-draft-token on every role.

➖ Check 12 (Append-only): N/A — The new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — Draft RadixArk/Muse-Spark-Open-DFlash2@46c90f41 is stored as BF16 with no quantization_config and no MoE layers. In pinned SGLang 797af717, online NVFP4 returns UnquantizedLinearMethod for non-FP8 linears and DFlashDraftModel.supports_quantization = False, so the draft serves BF16, as shipped. There is no draft quantization or dtype override in the recipes or logs, and SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is unset (default False).

✅ Check 14 (Pareto coverage): PASS — One curve (Muse Spark Open / agentic-coding / GB300 dynamo-sglang / FP4 / run 42374203350). From 14 raw bmk_agentic_musespark_* artifacts on the pinned image (the superseded attempt-1 1P2D c224 copy dropped), pareto_coverage gives 12 frontier points at P90 E2EL and 13 at P75.

✅ Check 15 (PR description): PASS — The body's per-point settings (HiCache sizes, schedule policies, running-request caps, intervals, draft backends), disagg topologies and node counts, shared env, image and Dynamo version, and removed-point dominance claims all match the pinned recipes and run evidence.

✅ Check 16 (NVIDIA overview): PASS — Muse-Spark-Open has a populated SGLang cell in the NVIDIA overview Framework Recipes table.

➖ Check 17 (Metrics publication): N/A — Only dynamo-sglang configs are affected, with no TRT-LLM worker, so the Dynamo TRT-LLM metrics controls do not apply.

✅ Check 18 (Recipe/image/sweep consistency): PASS — (a) Both recipes resolve and nothing fails to generate. (b) Master workers, TP/EP and node math match every override. (c) The image digest resolves. (d)/(e) No new orchestration pins or dispatch. (f) The changelog description is accurate (SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0 restates the pinned default False; the pull/XXX placeholder is accepted by the validator). (g) All 9 removed points are dominated in tput_per_gpu and mean_intvty by head points under the new keys; the facts list them as undominated only because they compare per config key. For example, TP4 c40 is dominated by TP4 c46, and TP8 c10/c14 by 1P4D c56. (h) No re-tuned deltas; the image moves from dev-muse-spark-open to the 797af717 nightly. (i) stream-interval: 20 on the agg, prefill and decode roles, confirmed in the logs.

Assessed commit: e465e9a2199e4e6a397cc7ba59db9a24761c6194.

@functionstackx functionstackx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved as requested by Oren in the linked thread.

中文

根据 Oren 在所链接讨论中的明确要求,提交批准。

Created with Perplexity

…ynamo-sglang-agentic-agg-disagg-1103 [skip-sweep]

Conflict in inferencex-e2e/perf-changelog.yaml only: kept main's entries and
re-appended this PR's entry at the end, with its pr-link set to #4757.
Benchmark results are reused from the pinned sweep run 42374203350.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh nvpohanh removed the full-sweep-fail-fast Full sweep with canary gate; first failure cancels the rest of that matrix (recommended) label Nov 10, 2026
@nvpohanh
nvpohanh merged commit 47cdc9d into main Nov 10, 2026
30 checks passed
@nvpohanh
nvpohanh deleted the musespark-fp4-gb300-dynamo-sglang-agentic-agg-disagg-1103 branch November 10, 2026 09:11
@github-actions

Copy link
Copy Markdown
Contributor

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category new duplicate skipped unmapped failed
benchmark 14 0 15 0 0
eval 13 13 0 0 0
trace 14 0 0 0 0
power 93 0 0 0 0

benchmarkFiles 15, rawBenchmarkRows 29, coveredByPerJobFiles 15, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 17147, gpuMetricSamplesNew 2350788, powerPublicationPoints 14, changelogEntriesWritten 1

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据,各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants